{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "Last updated: 19/03/2015 \n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -d -u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "hypothesis: $h_{\\theta}(x) = g(\\theta^T x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sigmoid function: $g(z) = \\frac{1}{1 + e^{-z}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$0 \\ge h_{\\theta}(x) \\le 1$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h_{\\theta}(x) = g(\\theta^T x) = \\frac{1}{1 + e^{-\\theta^T x}}$ can be interpreted as the posterior probability. For classification\n",
    "use step function:  \n",
    "$h_{\\theta}(x) \\le 0.5 \\rightarrow 0$   \n",
    "$h_{\\theta}(x) > 0.5 \\rightarrow 1$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$h_{\\theta}(x):  \\theta_1 x_1 + \\dots + \\theta_n x_n >  \\theta_0$\n",
    "\n",
    "equivalent to \n",
    "\n",
    "$h_{\\theta}(x) = -\\theta_0 + \\theta_1 x_1 + \\dots + \\theta_n x_n > 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear Regression Cost function (gradiend descent):\n",
    "    \n",
    "$J(\\theta) = \\frac{1}{2m} \\sum (h_{\\theta}(x^{(i)}) - y)^2$\n",
    "\n",
    "Now using  $\\frac{1}{1 + e^{-\\theta^T x}}$ (not convex anymore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1/(1 + np.exp(-z))\n",
    "\n",
    "assert(sigmoid(0) == 0.5)\n",
    "assert(sigmoid(10000) == 1.0)\n",
    "assert(sigmoid(-10000) == 0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAFqhJREFUeJzt3Xu0XGV9xvHvk4SAXAQkGCEJNwk0WZZwM4BYOZRbiJBA\n",
       "FUhQlheUuGo0rLaKQiupVit1UYFSY0BAq5Zgq0CQyKXKQaAKREPCJYFECOaCAQssEbkk+usfeycM\n",
       "JydnzpwzM+/svZ/PWrPmzMye4eE4Przn3Xu/WxGBmZmVw5DUAczMrHlc6mZmJeJSNzMrEZe6mVmJ\n",
       "uNTNzErEpW5mViJ1S13S1ZLWSXqwj20uk7Rc0mJJBzU3opmZ9Vd/RurXAJO29KKkycC+ETEWOAeY\n",
       "06RsZmbWoLqlHhF3Ac/1sckU4Fv5tvcCO0ka2Zx4ZmbWiGbMqY8CVtU8Xg2MbsLnmplZg5q1o1Q9\n",
       "HnvtATOzBIY14TPWAGNqHo/On3sdSS56M7MBiIieA+ctakapzwdmAvMkHQ48HxHrBhvM+iZpdkTM\n",
       "Tp2jDPy7rE9iR+CA/LY32UBuj/z+zcDTZNOwv4az9oRvLwCe7XF7Lr9/PoI/tv1foqAaHRDXLXVJ\n",
       "1wJHASMkrQIuBLYCiIi5EbFA0mRJK4AXgQ81HtvMOoGEyMr6QGBCfn8gWXE/CCwBfgX8gk0lzlMR\n",
       "rH/tM74zO+Lbn29zdMvVLfWImN6PbWY2J46ZtZvELsBksiPZjgFeBhYDDwDXAZ8FVnh0XQzNmH6x\n",
       "NLpTByiR7tQB2ikfje8PnExW5AcAPwFuAmZFsHaQ/4juQb7fBkHtukiGpPCculk6EocAZ5IV+RvI\n",
       "Snw+cEcEL6fMZlvWaHd6pG5WchJ/Afw9MA64CjgdeCDChx6XkUvdrITyKZbjgAvIThD8MvAfEbya\n",
       "NJi1nEvdrEQkhgAnkY3Mtwe+CFwXwYakwaxtXOpmJZCPzN8DfA7YQFbm10fwp6TBrO1c6mYFJ7E9\n",
       "2eqoBwOfAn7k+fLq8kUyzApMYjxwH9no/O0RLHChV5tL3aygJN4P3Al8JYIPRfCH1JksPU+/mBWM\n",
       "xBuAS8mW7zgmgiWJI1kH8UjdrEAkxgI/A3YADnWhW08udbOCkDgNuAeYC5wZwQuJI1kH8vSLWQFI\n",
       "zALOBSZHsDB1HutcLnWzDidxKvBp4B0RPJk6j3U2l7pZB5OYCFwBnOhCt/7wnLpZh5LYG7gBONtT\n",
       "LtZfLnWzDiSxM3Az8M8RzE+dx4rD66mbdRiJ4cAtwJIIzk2dx9JqtDtd6mYdJF+Y65vAjsB7fAk5\n",
       "80UyzIrtH4DxQJcL3QbCpW7WISTOAj4MHB7Bi6nzWDG51M06gMRRwMXA0RH8JnUeKy4f/WKWWL4e\n",
       "+neAsyJ4OHUeKzaXull65wM/jeDW1EGs+Dz9YpaQxH7AOcABqbNYOXikbpZIfvjiJcBFEaxNncfK\n",
       "wSN1s3ROAvYBTkkdxMrDpW6WgMQ2ZFcv+lgEr6bOY+Xh6RezND4FLIrgttRBrFy8TIBZm0nsCfwS\n",
       "OCSClYnjWIdrtDs9Ujdrv4uBS13o1gqeUzdrI4njgIOBs1JnsXLySN2sTfIldS8Dzo3gpdR5rJxc\n",
       "6mbt80ngCeCm1EGsvLyj1KwNJHYDHgSOiGB56jxWHN5RataZ/gW40oVurVa31CVNkrRM0nJJ5/Xy\n",
       "+ghJt0h6QNJDkj7YkqRmBSXxNuBY4Iups1j59Tn9Imko8CjZF3INcD8wPSKW1mwzG9g6Ij4raUS+\n",
       "/ciI2NDjszz9YpUkcQWwOoLPp85ixdPs6ZeJwIqIWBkR64F5wNQe2zwFvDH/+Y3A//UsdLOqktgF\n",
       "OA2YmzqLVUO949RHAatqHq8GDuuxzZXATyStBXYATm9ePLPC+yhwYwTrUgexaqhX6v05NOZ84IGI\n",
       "6JL0VuB2SRMi4oWeG+ZTNRt1R0R3v5OaFYzEVsDHgSmps1hxSOoCugb6/nqlvgYYU/N4DNlovdY7\n",
       "yHcARcSvJD0B7A8s7PlhETF7oEHNCuhU4IkIFqUOYsWRD3a7Nz6WdGEj7683p74QGCtpL0nDgTOA\n",
       "+T22WUa2IxVJI8kK/fFGQpiV1Cyy5XXN2qbPkXpEbJA0E7gVGApcFRFLJc3IX58LfAm4RtJisv9I\n",
       "fDoinm1xbrOOJnEoMBq4MXUWqxafUWrWAhLfBpZE8JXUWazYGu1Ol7pZk+VLAjwC7BPBc6nzWLF5\n",
       "mQCz9D4GzHOhWwoeqZs1kcTWwJPA0REsrbe9WT0eqZulNQ1Y7EK3VFzqZk0iIXwYoyXmUjdrniOB\n",
       "7YBbUgex6nKpmzXPLODfIvhT6iBWXd5RatYEEnsAi4C9Iths3SOzgfKOUrM0Pg58y4VuqXmkbjZI\n",
       "EtsBK4HDIrzukTWXR+pm7TcN+F8XunUCl7rZ4L0P+GbqEGbg6RezQcnXeXkY2D2Cl1PnsfLx9ItZ\n",
       "e50OzHehW6dwqZsNzjTg2tQhzDZyqZsNkMTewFuBn6TOYraRS91s4M4Avh/B+tRBzDZyqZsNnKde\n",
       "rOO41M0GQGIcsCtwd+osZrVc6mYDMw24zot3WadxqZs1KF833VMv1pFc6maNOwgYCixMHcSsJ5e6\n",
       "WeOmkV1Yuj2nY5s1YFjqAGZFIjGErNQnp85i1huP1M0acwTwuwgeSh3ErDcudbPGeAepdTSv0mjW\n",
       "TxLDgNXAOyNYkTqPVYNXaTRrnS5glQvdOplL3az/puOpF+twnn4x6weJrYG1wIQIVqfOY9Xh6Rez\n",
       "1jgeeMiFbp3OpW7WP9OBealDmNXj6RezOiS2A9YAYyN4JnUeqxZPv5g130nAz13oVgQudbP6puGp\n",
       "FyuIuqUuaZKkZZKWSzpvC9t0SVok6SFJ3U1PaZaIxLbAMcD81FnM+qPPBb0kDQUuB44lm1O8X9L8\n",
       "iFhas81OwL8DJ0TEakkjWhnYrM2OBRZG8GzqIGb9UW+kPhFYERErI2I92Z+gU3tscybw/YhYDRAR\n",
       "v21+TLNkpuBRuhVIvVIfBayqebw6f67WWOBNku6QtFDSWc0MaJaKxFDgZFzqViD11lPvz/GOWwEH\n",
       "k807bgv8TNLPI2L5YMOZJTYReDqCx1MHMeuveqW+BhhT83gMbHZG3SrgtxHxEvCSpJ8CE4DNSl3S\n",
       "7JqH3RHR3WhgszaaCtyYOoRVi6QussXjBvb+vk4+kjQMeJRsFL4WuA+Y3mNH6Z+R7Uw9AdgauBc4\n",
       "IyIe6fFZPvnICkXiEeCDEdyXOotVV6Pd2edIPSI2SJoJ3Ep2od2rImKppBn563MjYpmkW4AlwJ+A\n",
       "K3sWulnRSIwFdsIXl7aC8TIBZr2Q+FtgvwhmpM5i1eZlAsyaw4cyWiF5pG7Wg8QI4FfAyAheTp3H\n",
       "qs0jdbPBmwz82IVuReRSN9ucp16ssDz9YlZDYhtgHbCvl9q1TuDpF7PBORpY4kK3onKpm72ezyK1\n",
       "Qqu3TIBZZUgMIVvA6+jUWcwGyiN1s9ccArwQwWOpg5gNlEvd7DVT8NSLFZxL3ew1PpTRCs+lbgZI\n",
       "7A3sBvw8dRazwXCpm2VOBn4YwR9TBzEbDJe6WWYqnnqxEvAZpVZ5EjsDTwK7RfBi6jxmtXxGqVnj\n",
       "JgF3utCtDFzqZp56sRLx9ItVmsRwsgW8xkXwm9R5zHry9ItZY44CHnWhW1m41K3qfBaplYoX9LLK\n",
       "khBZqU9OncWsWTxStyqbAGwAHkkdxKxZXOpWZVOAGyNoz9ECZm3gUrcq86GMVjoudaskidHAXsDd\n",
       "iaOYNZVL3arqZGBBBBtSBzFrJpe6VZWnXqyUfEapVY7EG4E1wO4RvJA6j1lffEapWX0nAPe40K2M\n",
       "XOpWRb5snZWWp1+sUiSGkS3gNSGC1anzmNXj6Rezvr0TWOlCt7JyqVvVeAEvKzWXulVGzQJenk+3\n",
       "0nKpW5WMB7YCFqcOYtYqLnWrkinAfC/gZWVWt9QlTZK0TNJySef1sd3bJW2Q9FfNjWjWND6L1Eqv\n",
       "z1KXNBS4nOxq6+OB6ZLGbWG7i4BbAB+2aB1H4i3A/sCdqbOYtVK9kfpEYEVErIyI9cA8stFOT58A\n",
       "/ht4psn5zJrlJODWCF5NHcSsleqV+ihgVc3j1flzm0gaRVb0c/KnPF9pnchTL1YJ9Uq9PwV9CfCZ\n",
       "yE5NFZ5+sQ4jsR1wFPCj1FnMWq3ehafXAGNqHo+Bzc7EOwSYJwlgBHCipPURsdmoSNLsmofdEdHd\n",
       "aGCzATgOuD+C51IHMatHUhfQNeD397X2i6RhwKPAMcBa4D5gekQs3cL21wA3RcQPennNa79YEhJX\n",
       "A4sjuDR1FrNGNXXtl4jYAMwEbiW74vp1EbFU0gxJMwYX1az1JIaS7ST1fLpVgldptFKTOBKYE8EB\n",
       "qbOYDYRXaTR7vVPwAl5WIfV2lJoVlsQQ4Azg3amzmLWLR+pWZkcAv4vgwdRBzNrFpW5lNp3sLGiz\n",
       "yvCOUiul/LJ1a4AjI1iROo/ZQHlHqVmmC/i1C92qxqVuZeWpF6skT79Y6UhsTXYG9ARfYNqKztMv\n",
       "ZnAC8LAL3arIpW5lNA1PvVhFefrFSiVfZncNsF8ET6fOYzZYnn6xqjsJuNeFblXlUrey8dSLVZqn\n",
       "X6w0JHYEfg3sGcHzqfOYNYOnX6zKTgXucKFblbnUrUymAdemDmGWkqdfrBQkdgVWALtH8GLqPGbN\n",
       "4ukXq6r3Agtc6FZ1LnUrC0+9mOHpFysBidHAYrKpl1dS5zFrJk+/WBWdDtzgQjdzqVs5eJlds5xL\n",
       "3QpNYl9gD+CO1FnMOoFL3YruDOC/ItiQOohZJ3CpW2FJCDgTT72YbeJStyLrAgTckziHWcdwqVuR\n",
       "zQIui6A9x+WaFYCPU7dCktgHuI9sRUafRWql5ePUrSpmAle70M1ezyN1KxyJHYCVwMERPJk4jllL\n",
       "eaRuVfABsnXTXehmPQxLHcCsERJDgE8CZ6fOYtaJPFK3opkE/B64O3UQs07kUreimQVc6sMYzXrn\n",
       "HaVWGBLjyNZ42dMrMlpVeEepldkngStc6GZb1q9SlzRJ0jJJyyWd18vr75O0WNISSfdIOqD5Ua3K\n",
       "JHYmu7rRnNRZzDpZ3VKXNBS4nGwH1XhguqRxPTZ7HHhXRBwAfAG4otlBrfI+AtwcwVOpg5h1sv4c\n",
       "0jgRWBERKwEkzQOmAks3bhARP6vZ/l5gdBMzWsVJDCM7g/S9qbOYdbr+TL+MAlbVPF6dP7clZwML\n",
       "BhPKrIepwJoI7k8dxKzT9Wek3u/DYyQdDXwYOHILr8+uedgdEd39/WyrtFnApalDmLWDpC6yZaUH\n",
       "pD+lvgYYU/N4DNlovWeQA4ArgUkR8VxvHxQRsweQ0SpM4iBgb+AHqbOYtUM+2O3e+FjShY28vz/T\n",
       "LwuBsZL2kjSc7PJh82s3kLQH2f/p3h8RKxoJYFbHLOBrEaxPHcSsCOqO1CNig6SZwK3AUOCqiFgq\n",
       "aUb++lzgc8DOwBxJAOsjYmLrYlsVSIwkm0/fN3UWs6LwGaXWsSTmAK9GMCt1FrNUGu1Or9JoHUni\n",
       "YOBUoOc5EWbWBy8TYB0nX173cuCCCHrd6W5mvXOpWyc6i+yvyGtSBzErGs+pW0eR2BFYBkzxyUZm\n",
       "jXenS906isRXge0j+GjqLGadwDtKrbAk3ga8n2zhODMbAM+pW0eQEHAZ8I8RPJM6j1lRudStU5wG\n",
       "7AJ8PXUQsyLz9IslJ7E9cDFwZgQbUucxKzKP1K0TnA/cGcFdqYOYFZ1H6paUxH7AOYAvgWjWBB6p\n",
       "WzL5ztFLgIsiWJs6j1kZeKRuKZ0M7AOckjqIWVm41C0JiX3JLlA+LYJXU+cxKwtPv1jbSexCdh3b\n",
       "CyNeu8KLmQ2eS93aSmIb4Abg+gjmps5jVjZe+8XaJl9S97tkV9CaFsGfEkcy63he+8U62ReAPYFj\n",
       "XOhmreFSt7aQ+AjZRcuPiOCl1HnMysqlbi0ncTzwT8C7vFiXWWu51K2lJP4c+A7wnggeS53HrOx8\n",
       "9Iu1jMTuwA+Bc72ui1l7uNStJSR2A24GrojgP1PnMasKl7o1ncRfAr8AfgB8KXEcs0rxnLo1jcRQ\n",
       "4ALgY8BZEfw4cSSzynGpW1NI7Eq2Q3Qb4FCvumiWhqdfbNAk3gn8kmzK5RgXulk6HqnbgOXrof9d\n",
       "fvtwBDcnjmRWeS51GxCJNwPfAN4MTIzgycSRzAxPv1iDJEZJfBVYlt/e5UI36xwudesXib0k5gAP\n",
       "AgG8LYJP+wIXZp3FpW59kthf4ptkO0GfBfaP4G+8M9SsM3lO3TaT7wA9lGwH6NHAZcBbI3g+aTAz\n",
       "q8ulbsCmKxJ1AVPILgj9CvB14OwIfp8wmpk1wKVeYfkJQ+8mK/JjgCXATcDxwLII2nNZLDNrmrqX\n",
       "s5M0CbiE7BJk34iIi3rZ5jLgROAPwAcjYlEv2/hydglJDAfGAxOAA4HD8se3kxX5ggh+my6hmfWm\n",
       "qZezkzQUuBw4FlgD3C9pfkQsrdlmMrBvRIyVdBgwBzh8QOmt3yR1RUT35s8zBHgLsD9ZeR9IVuT7\n",
       "A48DD+S3fwDujuCVdmXuVFv6XdrA+PeZVr3pl4nAiohYCSBpHjAVWFqzzRTgWwARca+knSSNjIh1\n",
       "LchbWfnOy+2ANwG7wDvOkRgH7AGMqbnfHXgeWAEsAu4i+w/zQ76M3BZ1Ad2JM5RJF/59JlOv1EcB\n",
       "q2oeryb7s73eNqOBSpV6PkIeCmwFDAe27uV+48/bkhX0lm7bk5X3xtvO+f16ssMKn4UJ2wIvkP3u\n",
       "b8vvVwGrI3i55f/CZtaR6pV6f3eU9Zzv6fV9Ej/q52eoH8+pj+fUy+u1tyE190N6eW5oj/uezw3r\n",
       "5TYE2JDfXgFeze97+/kl4MVebs+SFfMLbCpvngWeA56rLWtpzuyIr83u9TdpZpVVr9TXkP1Jv9EY\n",
       "spF4X9uMzp/rhSY1Fq9wNhb8Nq34cKnnY13Yin9OFfl32Vz+faZTr9QXAmMl7QWsBc4ApvfYZj4w\n",
       "E5gn6XDg+d7m033ki5lZ6/VZ6hGxQdJM4FayaYerImKppBn563MjYoGkyZJWkE0hfKjlqc3MrFd1\n",
       "j1M3M7PiaOmCXpJOk/SwpD9KOrjHa5+VtFzSMknHtzJHGUmaLWm1pEX5rez7K1pC0qT8O7hc0nmp\n",
       "8xSdpJWSluTfyftS5ykSSVdLWifpwZrn3iTpdkmPSbpN0k71PqfVqzQ+CJwK/LT2SUnjyebnxwOT\n",
       "gK9J8oqRjQngXyPioPx2S+pARVNzct0ksu/idEnj0qYqvAC68u/kxNRhCuYasu9irc8At0fEfsCP\n",
       "88d9ammRRsSyiHisl5emAtdGxPr8xKYVZCc6WWO883lwNp1cFxHrgY0n19ng+Hs5ABFxF9nhy7U2\n",
       "ndyZ359S73NSjY535/WHRq4mO4nJGvMJSYslXdWfP8tsM72dOOfv4eAE8D+SFkr6aOowJVB7dv46\n",
       "YGS9Nwx6lUZJt5OtNdLT+RFxUwMf5T22PfTxu72AbI2dz+ePvwBcDJzdpmhl4e9c8x0ZEU9J2hW4\n",
       "XdKyfARqgxQRIanud3bQpR4Rxw3gbQ2csFRd/f3dSvoG2UqL1pj+nFxnDYiIp/L7ZyRdTzbF5VIf\n",
       "uHWS3hIRv5G0G/B0vTe0c/qldp5tPjBN0nBJewNjAe8pb0D+P/BGp5LtlLbGbDq5TtJwsp338xNn\n",
       "KixJ20raIf95O7J1+f29HJz5wAfynz8A3FDvDS29SIakU8kuhTYCuFnSoog4MSIekfQ94BGytVL+\n",
       "OnzAfKMuknQg2RTCE8CMxHkKZ0sn1yWOVWQjgeuVrWcxDPhuRNyWNlJxSLoWOAoYIWkV8Dngy8D3\n",
       "JJ0NrAROr/s57lIzs/LwseFmZiXiUjczKxGXuplZibjUzcxKxKVuZlYiLnUzsxJxqZuZlYhL3cys\n",
       "RP4fvgBtFiBT0CcAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107cb4978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(-10, 10, 0.5), [sigmoid(z) for z in np.arange(-10, 10, 0.5)])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost Function and Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$J(\\theta) = \\frac{1}{m} \\sum^{m}_{i=1} \\big[ -y^{(i)} log(h_{\\theta}(x^{(i)})) - (1-y^{(i)})log(1-h_{\\theta}(x^{(i)}))  \\big]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\frac{\\partial J(\\theta)}{\\partial \\theta_j} = \\frac{1}{m} \\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)}) x^{(i)}_j$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\theta_j := \\theta_j - \\alpha \\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)}) x^{(i)}_j$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J(\\theta) = \\frac{1}{m} \\sum^{m}_{i=1} \\big[ -y^{(i)} \\\n",
    "log(h_{\\theta}(x^{(i)})) - \\\n",
    "(1-y^{(i)})log(1-h_{\\theta}(x^{(i)}))  \\big]\\\n",
    "%regularize\\\n",
    "+ \\frac{\\lambda}{2m} \\sum_{j=1}^{n} \\theta^2_j$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do not regularize $\\theta_0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial J(\\theta)}{\\partial \\theta_0} = \\frac{1}{m} \\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)}) x^{(i)}_j \\\n",
    "\\quad \\quad \\text{for } j = 0$$\n",
    "\n",
    "$$\\frac{\\partial J(\\theta)}{\\partial \\theta_j} = \\frac{1}{m} \\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)}) x^{(i)}_j \\\n",
    "+ \\frac{\\lambda}{m} \\theta_j \\\n",
    "\\quad \\quad \\text{for } j \\ge 1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Sebastian Raschka 2015\n",
    "# mlxtend Machine Learning Library Extensions\n",
    "\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "class LogisticRegression(object):\n",
    "    \"\"\"Logistic regression classifier.\n",
    "\n",
    "    Parameters\n",
    "    ------------\n",
    "    eta : float\n",
    "      Learning rate (between 0.0 and 1.0)\n",
    "\n",
    "    epochs : int\n",
    "      Passes over the training dataset.\n",
    "\n",
    "    learning : str (default: sgd)\n",
    "      Learning rule, sgd (stochastic gradient descent)\n",
    "      or gd (gradient descent).\n",
    "\n",
    "    lambda_ : float\n",
    "      Regularization parameter for L2 regularization.\n",
    "      No regularization if lambda_=0.0.\n",
    "\n",
    "    Attributes\n",
    "    -----------\n",
    "    w_ : 1d-array\n",
    "      Weights after fitting.\n",
    "\n",
    "    cost_ : list\n",
    "      List of floats with sum of squared error cost (sgd or gd) for every\n",
    "      epoch.\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, eta=0.01, epochs=50, lambda_=0.0, learning='sgd'):\n",
    "        self.eta = eta\n",
    "        self.epochs = epochs\n",
    "        self.lambda_ = lambda_\n",
    "\n",
    "        if not learning in ('sgd', 'gd'):\n",
    "            raise ValueError('learning must be sgd or gd')\n",
    "        self.learning = learning\n",
    "\n",
    "\n",
    "    def fit(self, X, y, init_weights=None):\n",
    "        \"\"\" Fit training data.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Training vectors, where n_samples is the number of samples and\n",
    "            n_features is the number of features.\n",
    "\n",
    "        y : array-like, shape = [n_samples]\n",
    "            Target values.\n",
    "\n",
    "        init_weights : array-like, shape = [n_features + 1]\n",
    "            Initial weights for the classifier. If None, weights\n",
    "            are initialized to 0.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        self : object\n",
    "\n",
    "        \"\"\"\n",
    "        if not len(X.shape) == 2:\n",
    "            raise ValueError('X must be a 2D array. Try X[:,np.newaxis]')\n",
    "\n",
    "        if (np.unique(y) != np.array([0, 1])).all():\n",
    "            raise ValueError('Supports only binary class labels 0 and 1')\n",
    "\n",
    "        if not isinstance(init_weights, np.ndarray):\n",
    "        # Initialize weights to 0\n",
    "            self.w_ = np.zeros(1 + X.shape[1])\n",
    "        else:\n",
    "            self.w_ = init_weights\n",
    "\n",
    "        self.cost_ = []\n",
    "\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            if self.learning == 'gd':\n",
    "                y_val = self.activation(X)\n",
    "                errors = (y - y_val)\n",
    "                regularize = self.lambda_ * self.w_[1:]\n",
    "                self.w_[1:] += self.eta * X.T.dot(errors)\n",
    "                self.w_[1:] += regularize\n",
    "                self.w_[0] += self.eta * errors.sum()\n",
    "\n",
    "            elif self.learning == 'sgd':\n",
    "                cost = 0.0\n",
    "                for xi, yi in zip(X, y):\n",
    "                    yi_val = self.activation(xi)\n",
    "                    error = (yi - yi_val)\n",
    "                    regularize = self.lambda_ * self.w_[1:]\n",
    "                    self.w_[1:] += self.eta * xi.dot(error)\n",
    "                    self.w_[1:] += regularize\n",
    "                    self.w_[0] += self.eta * error\n",
    "\n",
    "            self.cost_.append(self._logit_cost(y, self.activation(X)))\n",
    "        return self\n",
    "\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"\n",
    "        Predict class labels for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Training vectors, where n_samples is the number of samples and\n",
    "            n_features is the number of features.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        class : int\n",
    "          Predicted class label.\n",
    "\n",
    "        \"\"\"\n",
    "        # equivalent to np.where(self.activation(X) >= 0.5, 1, 0)\n",
    "        return np.where(self.net_input(X) >= 0.0, 1, 0)\n",
    "\n",
    "\n",
    "    def net_input(self, X):\n",
    "        \"\"\" Net input function. \"\"\"\n",
    "        return X.dot(self.w_[1:]) + self.w_[0]\n",
    "\n",
    "\n",
    "    def activation(self, X):\n",
    "        \"\"\"\n",
    "        Predict class probabilities for X.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n",
    "            Training vectors, where n_samples is the number of samples and\n",
    "            n_features is the number of features.\n",
    "\n",
    "        Returns\n",
    "        ----------\n",
    "        int\n",
    "          Class probability.\n",
    "\n",
    "        \"\"\"\n",
    "        z = self.net_input(X)\n",
    "        return self._sigmoid(z)\n",
    "\n",
    "\n",
    "    def _logit_cost(self, y, y_val):\n",
    "        logit = -y.dot(np.log(y_val)) - ((1 - y).dot(np.log(1 - y_val)))\n",
    "        regularize = (self.lambda_ / 2) * self.w_[1:].dot(self.w_[1:])\n",
    "        return logit + regularize\n",
    "\n",
    "\n",
    "    def _sigmoid(self, z):\n",
    "         return 1.0 / (1.0 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iris example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD7CAYAAAChScXIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAFrFJREFUeJzt3XuQFeWdxvHnNxfAAeQmAgJKNgoY75cFa6PriutGsoAx\n",
       "kAuEZBMq2ewSYioVSYyprco/ajbEikGNYZNoVXQ1yeLGhCgxJsZokipSbPDKTS0vBJAgYBguM8zl\n",
       "3T/mjJwZztB9Tr/d57x9vp+qqTp9pufttwd95u3fefttc84JABCGhmp3AAAQH6ENAAEhtAEgIIQ2\n",
       "AASE0AaAgBDaABCQprQPYGbMKQSACjjnrP97qYd24dCfzuY4vsyeK61dU+1e+JfH88rjOUmcV2jS\n",
       "OC9bVepdyiMAEBBCGwACQmiXNGtLtXuQjjyeVx7PSeK8QpPdeVnaa4/0fBAZWk0bAKrNVlXxg0gA\n",
       "8O1Qw+DBt08/9dTNZzc2HjnBTMcEXK1yTq6ra9Dh11+f/nx7+2c3Sy3dcX+W0AYQpDFjvvz3s2ef\n",
       "NmX27Fv2n3ji2L2NjY3V7lJsXV1d2r9/d/PatffPWLv2yyfv2fOtJ+L+LDVtAEEaO3bPqQsWLN07\n",
       "atT4jpACW5IaGxs1atT4jgULlu4dO3bPqeX8LKENIEhmrmHQoCFB37w3aNAQZ+bKymFCGwACQmgD\n",
       "QEAIbQBIwd69O5u/+MXLL37f+06YvWjR+Ct/+tNvneKjXWaPAEAKbr31X85pbh7cff/9b/zyueee\n",
       "PPE//3PhzGnTZu6fPv2SA0naJbQB1J0vfGr6ZZ0H9w0qfq9p6Kgjt35381M+2j948K3GZ5/9zfjb\n",
       "bvvjEy0tI7pmzpy775xzLn/jF7/47qTp0y/ZnKRtQhtA7vQP5f6B3Hlw36A/DR15pPhnLuwX4km8\n",
       "9NKGoQ0Nje6d77zgUO97p5129v6NG38/JmnbhDaA3OkfyuUGctKR+KFD+5sGD27pLH6vpWV4Z1vb\n",
       "ocSZS2gDQD9JR+ItLSd2trf3DeiDB//aNGRI3yCvBKENoO40DR11pH8INw0ddWSg/ct1+ukXHOzu\n",
       "7rKXX97Q0lsiefXV50dMmjS9NWnbhDaA3Okfyv0D2dcHjgMZOnRk17nnXvHGPffcMO3GG3/87LPP\n",
       "/nbECy88Ne7mm3/9u6RtJwptM5ss6QeSTpbkJP2Xc25l0k4BQBJJQ9nHSPz663/w3C23fOi8RYvG\n",
       "/1NLy4gjH//4zc9Onz4z0XQ/KflIu0PS551zT5vZMEn/Z2aPOec2Je0YAFSLj5H4qFHjO77+9d+u\n",
       "99GfYonuiHTOveGce7rw+oCkTZK83PUDADiWt9vYzWyKpAskrfPVJgCgLy+hXSiNrJb0ucKIGwCQ\n",
       "gsSzR8ysWdKDku5zzj1Ueq/Zc4++nrVFWr416XEBIF9WTJUenxa1V9LZIybp+5I2OuduG3jPtWuS\n",
       "HAcA8m/51r4DWptTaq+k5ZF3S1os6Qoz21D4ujphmwCAASQaaTvnfifW5AaAzBC4ABAQQhsAAsLa\n",
       "IwDg2b33/seUJ5/88eRdu14ZfuGF79n+1a+uecZX24y0AdSd117TCb2v29tl27driM/2TzppctuC\n",
       "Bcu3XnLJNdt8tisR2gBy5sABNba2qrF3e9cuDerqOvr9PXvU/M1v6tynntKY9nbZypU68+c/1ySf\n",
       "fZg9+1/feM97Prlr2LDR3pZ77UVoA8iVX/1K47/xDZ3T2qrGV15Ry9e+pgs2bdLw3u+PGaOOpUv1\n",
       "/I9+pKnLlunShga5JUv0cnEbq1dr0pYtGiZJbW1quPtu/c3+/ZWUk13S0zkGNW0AuTJvnra/+aaG\n",
       "3HCDZjonzZ+vl84+W30ePnDaaTo0aJC62tvVdNFFerO5uW+6jh+vw9/+ts765Ce16eGHdeqQIeps\n",
       "aVEFT52xZCdTAqENIFcaGqRZs7Rz3bqeFUdnzNCe4u/3lkQmTlTre9+rjatW6azGRnVfdtnR/S69\n",
       "VHs6OtSwcqXOGz1ah2+6SeubKkpL/yNtyiMAcuWVV9Ry++0658Mf1paZM7Wjt1TS+/22NjWOG6dD\n",
       "112nzWeeqQNLl+r5ffs0uLiNtjY1rFuncY2N6j58WE0vv9xTKimf/5E2oQ0gV15/XS1z5uiVK6/U\n",
       "X5Ys0ctnnKF9u3cfDeURI9T5sY/p1d6SyNSpOjhvnnYUt/G97+n0IUPUeccd+v0HP6gX77pLZ731\n",
       "VvzKRGdnhw4fPtDQ3d1p3d1d1tZ2qKGzs8PL+VEeAZArl1+uN3tfNzRIixfrtXLbmD9fr40dq/am\n",
       "pp5SyTveoYMjR8avaX/nO9dNfeSRVVN7t9///mGT5sz59y1Ll975Yrl96Y/QBoB+JkxQe/H2xIlq\n",
       "K+fnly27a+uyZXelsgQ15REACAihDQABIbSBbAyXNLloe3LhPaAshDaQjZGSFkg6tfC1oPAeUBY+\n",
       "iASysU09z1JdXNi+r/AeKuScdR850maDBg3xfwdLRo4caTPnrLucn2GkDcC3TEpBu3ePeX316jtH\n",
       "7927s7mreEWoAHR1dWnv3p3Nq1ffOXr37jGvl/Oz5ly6f6TMzEnu06keBKh9k9VTEnmwsD1f0mrl\n",
       "c7Sd0bkeahg8+M5pkydvOqepqf0EsxRuP0yJc3KdnYMPb9t25nPt7Z/ZIrWUGG3bKufcMedEaAPZ\n",
       "GK6eGnZvcE2W9JbUdyGjHDlVfUtBZY0mIQ0U2pRHgGy0qu9Ic5v8B3aeZqjk6Vy8IrSB/KiVGSqT\n",
       "1VMSua/wNV99AziOWjmXmkN5BMiXWihL+CoF1cK5VBHlESBNXM4fFVUK4neVAKEN+FELl/M+yhJZ\n",
       "iPO7CuVcMkd5BPCn2pfzIc1QifpdhXQuKaE8AtSyCZIuLtq+uPBeiHyUPyixDIDQBvxIejk/UdJH\n",
       "JM0ofH2k8F45aqFEE6cfzC5JgPII4IePy/kZkj5aeH2vpD9W0I9ql2ji9IPZJbFQHgHSlPTmmeGS\n",
       "xhRtj1F1LvfjlB2SliayuNEotwhtoDb8naR5kh4pfM0rvFeOrMoOWZQ/otTt7BLKI0BtmCDpPEkX\n",
       "Frb/JOkZSTvLaCPLskMW5Y/jqYPZJaXLI6ynDdSGnZKadTS0N6q8wJZ6Aqs4tKq1gmAW/aiVc80c\n",
       "5RHUMh/TurKo0cYRNaUvzuV+Fr+POP2o29JELSC0Uct8TOvyUaP1IWpK31vqWXP69cLX6sJ7vvsZ\n",
       "1UacfsTZBymhpo1a52NaV9IarS+1MqUv51Pl8oIpf8inur0zroSsfhdRx+HfJEWENmpZnNqpj+ln\n",
       "WdRoL5a0UD0j7HsLry8+7k8cK6qfWS3EFHWcur1bMQuJyyNmdrekf5b0F+fcOSW+T3kElYo7rSvp\n",
       "9LMspo9NUE8Ne31h+2JJ2+V/Sl9WCzFFHYcSTGLpTfm7R9Ltkn7goS2gmI9pXXHayGL62E71Dej1\n",
       "A+0IHE/i8ohz7ilJ+zz0BahEPU0/81EK8lG6iDpOPf2bZM7L7BEzmyJpDeURVEEd3BnXh487EZOW\n",
       "LqKOU2//Jimp6h2Rs+cefT1ri7R8azbHRR2IKm3ECZCs6s1py+ouwajj1O3dismsmCo9Pi1qr4xC\n",
       "e+2abI4DHKO3HPBgYXu+em4GKQ6V3htfesuFCyX9t8oL7TjHSaq47FB8jHJC0UcbSMXyrX0HtDan\n",
       "1F6UR1AP4pQDauXGl+PxMZqvhSsCxJLSzTVm9oCkP0iaambbzOwTSdsEPBouaVzR9jiVXnvkeGtZ\n",
       "18rNIj7WoWYt68AlLo845xb66AiQkmmSFkl6oLC9SD2lj+Ipd71rWf+ssD1PUrekRwvbcUoflB2Q\n",
       "CdYeQd4NlzRd0lWF7cckbdaxH0RGrWXN08ORMdbTRn1qlbSraHuXjg3SPK1ljZxj7RGEzsc61VH7\n",
       "TFdPWaX3+4sK75WDRZbgBaGN0PlYpzpqn1I1xHLriiyyBC+oaSMPfEzXi5LFOtYssoQirKeN7NXC\n",
       "Jb+vR3RFTRsEMkFoI01ZXPJHrVPtow+90wZ/VfhaVHivHCyyBC8ojyBtaV/yx1k3xMcCSVHTBuO0\n",
       "wSJLKANT/pBPB3TslL4Dno8RZ9pgnDZYZAmJUR5BmrK45PexxnQUSheoGZRHkKasLvl9rDF9PJQu\n",
       "UAWUR5C9LC75S83s2Fd03KweWQZkgvIIQudjZgcQDEbaCN0WSffr6MyO+wvvAbnESBuhi5rZUQs3\n",
       "+ADeENoIXdTMDtb0QK5QHkHoehd76v1wsP9iT9vU8/CC4tklfJCIYBHaCB0zO1BXKI9gIHmpBce5\n",
       "MSYv54o6QGhjIHmpBcdZTzsv54o6QHkEA8lLLThO+SQv54o6wEgbAALCSBsDKa4Fq/C6eJZGntTT\n",
       "uSJwLBiFgdTTIkn1dK4IBo8bw1HMluirVX1H1dtEYKNGEdr1Kc5sCWZUADWImnZ9ijNbghkVQA1i\n",
       "pA0AAWGkXZ/izJZgRgVQg5g9Up/izJZgRgVQVTxuDEfFuUuQhZiAGkRNGwACQmgDQEAIbQAICKGN\n",
       "SnFXJVAFhDYqxR2TQBUwewSV4o5JoAoSj7TN7Goz22xmL5rZl3x0CgBQWqLQNrNGSXdIulrSuyQt\n",
       "NLMzfXQMNS/OsxcBeJZ0pD1D0kvOuVedcx2SfijpmuTdQgDiPHsRgGdJa9oT1beO+WdJMxO2iTBw\n",
       "xyRQBUlDO+bCJbPnHn09a4u0fGvC4wJAzqyYKj0+LWqvpKG9XcfO1f3zsbutXZPwOACQc8u39h3Q\n",
       "2pxSeyUN7fWSzjCzKZJ2SPqQpIUl9rshop2vJewHANSFRKHtnOs0s2WSHpXUKOn7zrlN/ff7upYP\n",
       "2Ma9+ujYPRq9slXDU1nys1UjvpJGuwBQDZmsp+2kqPW0l6Rx7Inadn6Hmjt8t7tb416QdLfvdgHg\n",
       "qNLraddKaKfpMp+NrdD1775NnzsljT8GkrRb476QRrsAQlO/oZ0Gr38Iep2rZ963R6Ob02i7VcNb\n",
       "KRUBISG069llE7XtA2k1vkOTnhblIsAzQhvpWDJR285Pq/EdmvQ/kp5Kq32gdhHaCMw1euim9boo\n",
       "lTW6O9TcsVvjHkqhaf7AwBNCG3jbuXrmVt9t7tHo5kKp6EXfbYs/BnWI0AbStuRcPXNWGg0X/UFI\n",
       "A59H1CRCGwjWNXropjTaXa+LhhdKRS+k0b74g5AAoQ2ghDT/IOzQpB1ptK26WPqC0AaQrSUrdP3J\n",
       "vhstLH3RnP8/CIQ2gPxIbekLqedmNJ/ttmrEyyq7VERoA0AcXv8grND1J1ey9MVujR9GaANAdZS9\n",
       "9IVJi0uFdtL1tAEA0SqZZ7+41JtJH+wLAMgQoQ0AASG0ASAghDYABITQBoCAENoAEBBCGwACQmgD\n",
       "QEAIbQAICKENAAHhNvacmibd2CW9/XzFRql1i3RzaMcA0BehnVNd0vCXpLeXlzy9KFxDOgaAviiP\n",
       "AEBACG0ACAjlkZxqlFpP71dvDvEYAPriIQgAUINMKvm4MUbaqFgWs0fGSrc3Ff132il17pY+6/MY\n",
       "EjNhEA5CGxXLYvZIk9S0U+rs3Z6Q0n+zzIRBKPggEgACwkg7QHEu5X2UFUZJ3xksvV1Ta5fcPunf\n",
       "erffkk68QDqxaBtAygjtAMW5lPdRVhgs2RvS259Ujy8KcElqlnRf0Xv/WLSvL51S54R+f3x8H0Ni\n",
       "JgzCQWijYkOl/WcVhdvQFOrAaXzoWAofOiIU1LQBICAVj7TN7AOSvippuqS/dc79yVen6l1UzTrO\n",
       "pXxUWSGqXl14r09JpL3fMfZIo8dLowf6vhRdW486V6b8AX0lKY88J+laSas89QUFUTXrOGESFWxR\n",
       "9WpJGiPtPV4/BkuKaiOqth51rkz5A/qq+H8A59xmSTI75v9TAEBK+CDSMx+X2XukUROLyg6HKpiV\n",
       "Eaes0F1iZFxsrzTqvKJ+7C3Rj66INrqlho09E01iHbMSlDZQT44b2mb2mKTxJb51o3NuTdyDzJbm\n",
       "9r6eJW1ZLm2N38Ww+LjMHizZS0Xb76gg6KLKCu2STum33V+zZA8UBfUV/frRLmliRBsdkltY9HMd\n",
       "/YI/qj4fZ8qfj985U/5QbSukqY9L06L2O25oO+eu8tGZtVLsgIdkUvcJReFkKVwRRdWrJalB6n5X\n",
       "UT8a+vVjiNQdVW8eLe17JkF9nil/qBfLpa3FA1qT5pTaz1cYUNguiFPaSHo5n1U5IIvSRpQ458qd\n",
       "magnSab8XStppaSTJD1sZhucc7O99SxQcUobUZfzUSWBOOWAqDbilAOiShtxShdJyw5xzjWLOzOB\n",
       "WpFk9shPJP3EY19ywUdpw0dJIKqNOCPzqNJGnH5mUXbI4s5MoFYwewQDOiid+II0omi7bkewzFBB\n",
       "rSC0PcuiZJDVTIcOSYuLgrojjYNEiHOuWfw+uPkGtYLQ9iyLkkFWI7yR0v4NVQ6qOOfKiBf1hNAu\n",
       "E5fJAKqJ0C5TPV0mc8PJUfwuUCsIbQyIK4ij+F2gVrCeNgAEhJF2mbK4TKZuDmAghHaZsgjPeqqb\n",
       "AygP5REACAihDQABoTxSg5heBmAghHYN4kNHAAOhPAIAASG0ASAghDYABITQBoCAENoAEBBCGwAC\n",
       "QmgDQEAIbQAICKENAAEhtAEgIIQ2AASE0AaAgBDaABAQQhsAAkJoA0BACG0ACAihDQABIbQBICCE\n",
       "NgAEhNAGgIAQ2gAQEEIbAAJCaANAQAhtAAhIxaFtZivMbJOZPWNm/2tmI3x2DABwrCQj7V9KOss5\n",
       "d56krZK+7KdLAICBVBzazrnHnHPdhc11kib56RIAYCC+atpLJD3iqS0AwACajvdNM3tM0vgS37rR\n",
       "ObemsM9XJB1xzt0/UDuzpbm9r2dJW5b3lFMAAAUrpKmPS9Oi9jPnXMUHMbOPS/qUpCudc20D7OOc\n",
       "9OmKDwIAdcikVc456//+cUfax23Q7GpJyyVdPlBgAwD8SlLTvl3SMEmPmdkGM/u2pz4BAAZQ8Ujb\n",
       "OXeGz44AAKJxRyQABITQBoCAENoAEBBCGwACQmgDQEAI7RJWSFOr3Yc05PG88nhOEucVmizPi9Au\n",
       "Ic6tpCHK43nl8Zwkzis0WZ4XoQ0AASG0ASAgiRaMinUAs3QPAAA5VWrBqNRDGwDgD+URAAgIoQ0A\n",
       "ASG0S8jrk+bN7ANm9oKZdZnZhdXuT1JmdrWZbTazF83sS9Xujw9mdreZ7TKz56rdF5/MbLKZ/abw\n",
       "39/zZnZdtfvkg5kNMbN1Zva0mW00s1vSPiahXVpenzT/nKRrJT1Z7Y4kZWaNku6QdLWkd0laaGZn\n",
       "VrdXXtyjnnPKmw5Jn3fOnSXpEkmfycO/V+EBMFc4586XdK6kK8zs0jSPSWiXkNcnzTvnNjvn8vJ8\n",
       "zhmSXnLOveqc65D0Q0nXVLlPiTnnnpK0r9r98M0594Zz7unC6wOSNkk6pbq98sM5d6jwcpCkRkl7\n",
       "0zweoR2NJ83XpomSthVt/7nwHmqcmU2RdIF6BkTBM7MGM3ta0i5Jv3HObUzzeBU/uSZ0vp40X2vi\n",
       "nFdOMFc1QGY2TNJqSZ8rjLiDV7gqP7/w2dejZvYPzrkn0jpe3Ya2c+6q432/8KT590q6MpMOeRJ1\n",
       "XjmyXdLkou3J6hlto0aZWbOkByXd55x7qNr98c0591cze1jSxZKeSOs4lEdKKHrS/DU5ftL8MXda\n",
       "BWa9pDPMbIqZDZL0IUk/q3KfMAAzM0nfl7TROXdbtfvji5mdZGYjC69PkHSVpA1pHpPQLi2XT5o3\n",
       "s2vNbJt6Pr1/2MzWVrtPlXLOdUpaJulRSRsl/cg5t6m6vUrOzB6Q9AdJU81sm5l9otp98uTdkhar\n",
       "Z3bFhsJXHmbJTJD0eKGmvU7SGufcr9M8ILexA0BAGGkDQEAIbQAICKENAAEhtAEgIIQ2AASE0AaA\n",
       "gBDaABAQQhsAAvL/YKKQDvfVyVsAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10bccfb38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import sys\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "sys.path = ['/Users/sebastian/github/mlxtend/'] + sys.path\n",
    "from mlxtend.evaluate import plot_decision_regions\n",
    "\n",
    "\n",
    "df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)\n",
    "\n",
    "# setosa and versicolor\n",
    "y = df.iloc[0:100, 4].values\n",
    "y = np.where(y == 'Iris-setosa', 0, 1)\n",
    "\n",
    "# sepal length and petal length\n",
    "X = df.iloc[0:100, [0,2]].values\n",
    "\n",
    "# standardize features\n",
    "X_std = np.copy(X)\n",
    "X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()\n",
    "X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()\n",
    "\n",
    "lr = LogisticRegression(eta=0.1, epochs=100)\n",
    "lr.fit(X_std, y)\n",
    "\n",
    "plot_decision_regions(X_std, y, clf=lr, res=0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEZCAYAAAB1mUk3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4HVWZ7/HvjySEDEBAMECAQCOTiDLIJCC5iBgVBbpF\n",
       "wKsi9EVth0ZAFNDbpvVKY3NVbG21GwRxAAcmwQeRQaIoyCQEZJBRAoQkAoGEISQkb/+x1iaVzd4n\n",
       "Jyd7V9U++/d5nnp2zfVWnaTeWmvVoIjAzMz622pVB2BmZtVzMjAzMycDMzNzMjAzM5wMzMwMJwMz\n",
       "M8PJwDpE0l8lvaWE7XxH0ue7vZ2hysdh39x/sqQzqo7JbDCcDIY5SXtJuk7S05KelPR7SW/M0z4k\n",
       "6doObSpy1zGt4ouIf4qI/zfE9R0m6QZJz0qaI+mPkv6pM9EuC/HlnohTIuLoVV2hpM0kLZXU9v+r\n",
       "pGmSFkuan7u/SPqmpA1Wdfvdkvfp76qOwxIng2FM0lrAL4FvAOsAk4B/BV6sMq4qSDoeOB34CjAx\n",
       "IiYCHwX2lLR6m2Xq9v9DA0wL4LyIWIv0tz4Y2AC4pc4JgYH3ycoUEe6GaQe8EZjXZtq2wAvAS8AC\n",
       "4Kk8fm3gB8Bc4K/A5wAVljsauAuYD9wJ7JDHPwQcD8wAngZ+AozO0yaQktJc4CngUmBSYZ0fAh7I\n",
       "63wQeB+wDbCwRXzfB75UWPZA4DbgGeB+4G0t9nVt4Fng4BUcr+8D3wEuy/PvC7wTuDWvfybwhaZl\n",
       "PgA8DDwBnJyPw7552jTgh4V5dweuA+blmPcpTJsOfBH4fT4OvwZelafNBJbm47AA2K1F7MttK49b\n",
       "LW/ntMK4A/K4ecAfgO0L0z4LPJq3f09hP0bkfbs/T7sZ2DhP2wa4EngyL3NI0/H8z/y3nw/8Efi7\n",
       "PO13eZ+ezft0SKu/ibsSzxdVB+Cui39cWDOfpL4PTAXWaZp+BHBt07gfABcB44DJwF+Ao/K0Q/LJ\n",
       "Yuc8vAWwae7/a/7PvgHpyvQu4CN52rqkK9U1gPHAz4CL8rRx+US7ZR6eCLx2gPjOBr6Y+3clJZ63\n",
       "5OGNgK1bHIepwGJgtRUcr+/n9e2Rh0cD+wDb5eHtgdnAgXn4tflEthewOvDVvJ3GSfQL5BM0qVT2\n",
       "BDA1D++Xhxsn/OnAfcBr8nG6Bvi3PG1yPnG2jZ8WySCP/1fgj7l/R2AOsAvpivyDpOQ1CtialHQ2\n",
       "yPNuyrIT9wnA7YW/0fb5bzoOeCT/nVYDdgD+BmxbOJ5PkC5KRgA/IpVeGrEtbWzDXfVd3YrB1kER\n",
       "0ThRBXAGMFfSLyS9Os+yXBFd0gjgUOCkiHguIh4mneA+kGf5P8BXIuKWvP4HImJmY3PAf0TE7IiY\n",
       "R7r63yHP91REXBQRCyPiWeAU0km2YSmwvaQxETEnIu5qFV8L/wh8LyKuztuZFRF/aTHfesATEbG0\n",
       "sK/XSZon6XlJexXmvTgirs/rezEifhsRd+bhO0glnkbs7wEujYjfR8Qi4P/mfXl5M4X+9wOXRcTl\n",
       "eV1Xka6w35mnB3B2RNwfEQtJCXOHQR6HgTxOOnEDfBj4r4i4KZIfkKoM9yCVwEYD20kaFREzI+LB\n",
       "vNw/Ap+LiPsaxyEiniKVMh6KiHMiYmlE3AZcSLpoaLgwIm6OiCXAjwv7ZDXjZDDMRcQ9EXFkRGwC\n",
       "vI509Xx6m9nXI10lPlwYN5N0VQuwMak6p53Zhf4XSKUAJI2V9F/5TptngN8Ca0tSRDxHSkAfBWZJ\n",
       "+qWkrQe5eyuKp+FJYL1iG0BEvCki1snTGuODdKX7Mkm7SbpG0lxJTwMfAV6VJ29EKik11vl8Xl8r\n",
       "k4FDcgKaJ2kesCepJNXQ8vitokmFmCYDxzfFsDGwYUQ8AHyKVMKYI+k8SRvm5Tah9XGeDOzWtL73\n",
       "kUp3kI7nnC7sk3WBk0EfyVfN55CSArzy7p8nSNUcmxXGbcqyE94jpGqMlXU8sBWwa0SsTbqyVu6I\n",
       "iCsiYn/SifEeUimmVXzNBhvP9aQr4INWPnTOBS4m1ZFPAL7Lsiv1WaQTJZCSHssSRbOZpGqcdQrd\n",
       "mhHx74OIYTB3ab1inpz83gU07siaCXy5KYbxEfFTgIg4LyL2Jp3kg9TYDu2P80zgty326eODiNdq\n",
       "xslgGJO0taTjJE3Kw5sAh5NOjpCu2jaWNAogF+V/BnxZ0nhJk4FjSXW9AGcCn5a0k5LXSNp0EKGM\n",
       "J10VPiNpXVJdeiPGV0s6UNI4UiJ6DljSKr7GIiw7GX8POFLSvpJWkzSpVakiIp4m1Z1/W9I/SFoz\n",
       "z78Dqd67uO5Wsc+LiEWSdiVd+TZcABwgqXFH0hdp/3/qR8C7JO0vaYSkNSRNafxtBtg+pHr4paQ2\n",
       "mnZeXlbSSEnbAucBrwa+liedAXxU0q757zdO0jvz33qrfBxHkxLnQpb9Hc4EvpT/3pL0+vx3/CWw\n",
       "laT3SxqVu10kbbOC/WmYs4J9shI5GQxvC4DdgBskPUtKAreTrtQBribdETRb0tw87pOkE/KDpCvK\n",
       "H5MabYmI84Evk66W55Pqh9dps+3icwenA2NIJY/rgF8Vpq1GSjiPkaoz9gYa9/63iu/l9UbETcCR\n",
       "wNdJDb/XkEoyrwwm4jTgOOAzpOqY2aSr/M+wLDm2elbiY8AXJc0ntQn8tLDOO4GP5+Mxi3SnVLGa\n",
       "qRjro6Q7n04m3VU1k/R3UNP8rZZ9nnTc/5CrY3ZttYvAoZIW5GPxC1IS2TkiZuf13EK6G+xbOdb7\n",
       "SI3IkNoL/i0v8zipyvCkPO1rpIuEK0iN/WcAa+T2n/2Bw0h/v8fzOhq36rY6nsXhacA5eZ/e02Kf\n",
       "rESK6M7HbSSdRWocmxsR2zdNOx44DVgvN0SZmVmFulkyOJt0S99yclXFW1m+kdLMzCrUtWQQEdeS\n",
       "Hmxp9jVS0dzMzGqi1DYDSQcCj0bE7WVu18zMBjayrA3l2+5OJlURvTy6rO2bmVl7pSUD0i1kmwEz\n",
       "JEF62OUWSbtGxNzijJK606ptZjbMRcSQLrJLSwb5Uf7Gk4lIeoh021vLu4mGukPDjaRpETGt6jjq\n",
       "wMdiGR+LZXwsllmVC+mutRlIOo90T/lWkh6RdGTTLL76NzOria6VDCLi8BVM90ctzMxqwk8g19/0\n",
       "qgOokelVB1Aj06sOoEamVx3AcNC1J5BXRXqZpdsMzMxWxqqcO10yMDMzJwMzM3MyMDMznAzMzAwn\n",
       "AzMzw8nAzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAyMDMznAzMzAwn\n",
       "AzMzw8nAzMyocTKQ8GcvzcxKUttkAIyuOgAzs35R52QwruoAzMz6RVeTgaSzJM2RdEdh3GmS7pY0\n",
       "Q9KFktZus/j4bsZmZmbLdLtkcDYwtWncFcB2EfEG4F7gpDbLumRgZlaSriaDiLgWmNc07sqIWJoH\n",
       "bwA2brO4k4GZWUmqbjM4CriszTRXE5mZlWRkVRuW9DlgUUSc23qOPY+WrtsnD0yPiOklhWZm1hMk\n",
       "TQGmdGRdEdGJ9bTfgLQZcGlEbF8Y9yHgaOAtEbGwxTIBcXgEP+lqcGZmw4ikiIghPaNVeslA0lTg\n",
       "BGCfVomgwG0GZmYl6fatpecB1wFbS3pE0lHAN0ntAVdKulXSt9ss7mRgZlaSrpYMIuLwFqPPGuTi\n",
       "bkA2MytJ1XcTDcQlAzOzktQ5GbhkYGZWkjonA5cMzMxK4mRgZma1TgauJjIzK0mdk4FLBmZmJalz\n",
       "MnDJwMysJHVOBi4ZmJmVxMnAzMxqnQxcTWRmVpI6JwOXDMzMSlLnZLC6xIiqgzAz6wd1TgbP4dKB\n",
       "mVkpnAzMzKz2ycCNyGZmJahzMngWlwzMzEpR52TgkoGZWUnqnAxcMjAzK0mdk4EbkM3MSlL3ZOBq\n",
       "IjOzEtQ5GbiayMysJHVOBi4ZmJmVpM7JwCUDM7OSdC0ZSDpL0hxJdxTGrSvpSkn3SrpC0oQBVuEG\n",
       "ZDOzknSzZHA2MLVp3InAlRGxFXB1Hm7H1URmZiXpWjKIiGuBeU2j3w2ck/vPAQ4aYBWuJjIzK0nZ\n",
       "bQYTI2JO7p8DTBxgXpcMzMxKMrKqDUdESIr2c+x+EOz8eunb04DpETG9pNDMzHqCpCnAlI6sK2KA\n",
       "8/GqrlzaDLg0IrbPw/cAUyJitqQNgWsiYpsWywXE3sCpEezVtQDNzIYRSRERGsqyZVcTXQIckfuP\n",
       "AC4eYF5XE5mZlaSbt5aeB1wHbC3pEUlHAqcCb5V0L7BvHm7HDchmZiXpajXRUOVqoknALRFsWHU8\n",
       "Zma9oJeqiVaGSwZmZiWpczJ4DhgnMaQsZ2Zmg1fbZBDBEmAxsEbVsZiZDXe1TQaZq4rMzEpQ92Tg\n",
       "20vNzEpQ92TgkoGZWQnqngz8GmszsxL0QjJwNZGZWZfVPRm4msjMrAR1TwYuGZiZlaDuycAlAzOz\n",
       "EtQ9GbhkYGZWgl5IBi4ZmJl1Wd2TgauJzMxKUPdk4GoiM7MS1D0ZuGRgZlaCuicDlwzMzErQC8nA\n",
       "JQMzsy6rezJwNZGZWQnqngxcTWRmVoK6JwOXDMzMSlD3ZOCSgZlZCXohGbhkYGbWZZUkA0knSbpT\n",
       "0h2SzpU0us2sriYyMytB6clA0mbA0cBOEbE9MAI4rM3sLwIjJUaVE52ZWX+qomQwH1gMjJU0EhgL\n",
       "PNZqxggCeBJYr7zwzMz6T+nJICKeAr4KzARmAU9HxFUDLPIYMKmM2MzM+tXIsjcoaQvgU8BmwDPA\n",
       "zyX974j4cdN801LfR8fBOvvDKTeXG6mZWb1JmgJM6cS6Sk8GwBuB6yLiSQBJFwJvApZLBhExLU1n\n",
       "Q2BeyTGamdVeREwHpjeGJX1hqOuqos3gHmB3SWMkCdgPuGuA+V1NZGbWZVW0GcwAfgDcDNyeR//3\n",
       "AIs4GZiZdZkiouoYXkFSRIRSP1OBYyN4W8VhmZnVWvHcubLq/gQypDuOXDIwM+uiXkgGriYyM+uy\n",
       "FSYDST8czLguegoYIzG2xG2amfWVwZQMXlccyE8N79ydcF4pP4U8C9iorG2amfWbtslA0smSFgDb\n",
       "S1rQ6IC5wCWlRZi4qsjMrIvaJoOIOCUi1gT+f0SsWejWjYgTS4wRUjJwycDMrEsGU030S0njASR9\n",
       "QNLXJE3uclzNfEeRmVkXDSYZfAd4XtIbgOOAB0kPjZXJ1URmZl00mGTwUkQsBQ4C/jMivgWs2d2w\n",
       "XsHVRGZmXTSYF9UtkHQy8H5gb0kjoPSPzbhkYGbWRYMpGRxK+uLYURExm3RSPq2rUb2S2wzMzLpo\n",
       "UO8mkrQBsAsQwI0RMberQTW9X0NiDOk11mPycwdmZtakq+8mkvRe4AbgEOC9wI2SDhnKxoYqgheA\n",
       "54BXlbldM7N+MZg2g88DuzRKA5LWB64Gft7NwFpoVBU9UfJ2zcyGvcG0GQj4W2H4yTyubG5ENjPr\n",
       "ksGUDC4Hfi3pXFISOBT4VVejas23l5qZdUnbZCBpS2BiRJwg6R+APfOk64BzywiuiUsGZmZdMlA1\n",
       "0enAfICIuCAijouI44CLga+XEVwT315qZtYlAyWDiRFxe/PIPG7z7oXUlquJzMy6ZKBkMGGAaWt0\n",
       "OpBBcDWRmVmXDJQMbpb04eaRko4GbuleSG25msjMrEvaPoGcnzq+CFjEspP/zsBo4OCIeLxrQbV4\n",
       "ik5iNWAhsGYEL3Zr22ZmvWpVnkAe8HUUkgT8L9KnLwO4MyJ+M6QoVyaoNjskMRPYO4KHux2DmVmv\n",
       "WZVkMOBzBpEyxW9y1zGSJgBnAtuRksxREfHHQSzaqCpyMjAz66DBPHTWDd8ALouI90gaCYwb5HJu\n",
       "RDYz64LSk4GktYG9I+IIgIh4CXhmkIvfD2zZrdjMzPrVYN5N1GmbA3+TdLakP0k6Q9LYQS57O/CG\n",
       "LsZmZtaXqqgmGgnsBHwiIm6SdDpwIvAvxZkkTSsMTo+I6aRk8PmS4jQzqzVJU4ApHVnXYD5u00n5\n",
       "ltXrI2LzPLwXcGJEHFCYp93dRKNIVUqvyt84MDOzrKsft+m0/OnMRyRtlUftB9w5uGVZDNxLugvJ\n",
       "zMw6pKq7iT4J/FjS6sADwJErseztwOuBm7sRmJlZP6okGUTEDNI3lYdiBm5ENjPrqCruJlpVjZKB\n",
       "mZl1SM8mA6mST2+amQ1LPZcMIpgDLMZPIpuZdUzPJYPMVUVmZh3kZGBmZj2bDHxHkZlZB/VqMnDJ\n",
       "wMysg0p/HcVgrOiRaonVgaeBdSNYWF5kZmb11VOvo+iECBaRXmf92qpjMTMbDnoyGWSuKjIz65Be\n",
       "TgYzcDIwM+uIXk4GtwFvrDoIM7PhoCcbkNM8jANmAxtFsKCcyMzM6qvvGpABIngOuBHYp+pYzMx6\n",
       "Xc8mg+wKYP+qgzAz63VOBmZm1vPJYAawrsTkqgMxM+tlPZ0MIlgKXAm8tepYzMx6WU8ng8xVRWZm\n",
       "q6hnby1dNi8bAX8G1o9gSXcjMzOrr768tbQhglnAY/gBNDOzIev5ZJD9GlcVmZkN2XBJBm43MDNb\n",
       "BZUlA0kjJN0q6dIOrO5aYAeJdTqwLjOzvlNlyeAY4C5glVuwI3gBuAw4bFXXZWbWjypJBpI2Bt4B\n",
       "nAkMqeW7hbOAozq0LjOzvlJVyeDrwAnA0g6u8ypgA8nfODAzW1kjy96gpAOAuRFxq6QpA8w3rTA4\n",
       "PSKmD7TeCJZIfB84Ejh21SM1M6u3fA6d0pF1lf3QmaRTgA8ALwFrAGsBF0TEBwvzDOnBCYktgOuB\n",
       "jfN3ks3M+saqPHRW6RPIkvYBPh0R72oaP/QdEtOBb0ZwQQdCNDPrGb3+BHKns5Ebks3MVlLPv5vo\n",
       "lcsyDngUeF0Ej3U2MjOz+ur1kkFH5c9h/gz4SNWxmJn1imFXMkjLswVwA7BlBPM6F5mZWX25ZNAk\n",
       "ggeAXwDHVR2LmVkvGJYlg7QONgduBraK4MnORGZmVl8uGbQQwUPA+cDxVcdiZlZ3w7ZkkNbDpsCt\n",
       "wNYRPLHqkZmZ1ZdLBm1EMBP4KfCZqmMxM6uzYV0ySOtiEnAb8OYI7u7EOs3M6sglgwHkB8+mAWdK\n",
       "w39/zcyGol9Ojt8hvfbiY1UHYmZWR8O+mmjZOtkG+D2wU25LMDMbVlxNNAgR3AOcDnxX6tjX1czM\n",
       "hoW+SQbZvwMbAP9cdSBmZnXSN9VEy9bN5qQP4BwWwfRubMPMrAquJloJ+cnk9wPn5YfSzMz6Xt8l\n",
       "A4AIrgK+ClwoMabqeMzMqtZ31UTLtoGAHwOjgUMjeKmb2zMz6zZXEw1BBAEcCYwFzpEYUXFIZmaV\n",
       "6dtkABDBi8Dfk+4wOsNPKJtZv+r7k18ELwDvBrYEvuMSgpn1o75PBvDyd5PfSUoIF0iMrTgkM7NS\n",
       "ORlkEcwHpgLPANMlJlYckplZaZwMCiJYBHwIuAy4XmKHaiMyMytHJclA0iaSrpF0p6Q/S6rN6yEi\n",
       "iAimAScDV0p8zO8yMrPhrpLnDCRtAGwQEbdJGg/cAhwUEXfn6V1/zmAwJLYEfgY8ABwdwbyKQzIz\n",
       "a6vnnjOIiNkRcVvufxa4G9ioilgGEsF9wB7ALOAOiYMqDsnMrCsqfwJZ0mbAb4HtcmKoTcmgSOLN\n",
       "wBnAHcAnI3i84pDMzJazKufOkZ0OZmXkKqLzgWMaiaAwbVphcHpETC8xtFeI4HcSbwA+TyolnAZ8\n",
       "I4KFVcZlZv1L0hRgSkfWVVXJQNIo4JfAryLi9KZptSsZFElsBXwF2JHU0PyTCJZWG5WZ9btVOXdW\n",
       "1YAs4BzgyYg4tsX0WieDBol9gNOAMcCXgAsiWFJtVGbWr3oxGewF/A64nfSheoCTIuLyPL0nkgG8\n",
       "/PbTtwP/AqwFnAr8NL/3yMysND2XDFakl5JBQ04K+wEnAK8Dvgt8N4K5lQZmZn2j524tHY7yw2pX\n",
       "RrA/sD+wMXCvxE8l3uo3oppZnblk0EUSE4D3AUcDE4AfAedGcHelgZnZsORqoprLVUg7kRLDYcBc\n",
       "4CfARRHcW2VsZjZ8OBn0kPy9hH2A9wAHA08CF5Fus73ZdyOZ2VA5GfSo3I6wG3AQ8A7SF9cuB64A\n",
       "ro5gVoXhmVmPcTIYJiQ2JSWF/YB9gdnAb0i34f4ugtkVhmdmNedkMAzl6qQdSElhb2Av4AngOuD6\n",
       "3N3paiUza3Ay6AO5Sul1pLeo7gG8CdgQuA24CbgZuBW41wnCrD85GfQpiXWAnYFd8u+OwETSm1Vv\n",
       "L/z+OYKnqorTzMrhZGAvk1ibVL20faHbDngBuBO4C/gLcE/+fcwv2TMbHpwMbED5OYdJwGtztzWw\n",
       "Te7WBu4H7su/D5K+7PYg8EgEi6uI2cxWnpOBDZnEmsBrgC2BLZq6icDjwEPAw4VuJvAIKVk8X0HY\n",
       "ZtaCk4F1hcTqwCbA5sBkYNPC7ya5exZ4DHg0/z5G+kzoLFIieRyY60Zts+5zMrBK5Oqn9UlVUJNI\n",
       "L+fbKHeTSHc7bQisS3rSenbu5uRubqH7W+PXX48zGxonA6s1iVHAq0nVThNJT1o3hl+du/Xz73rA\n",
       "ItIzFY3uycLvU4XfRjcPeMYN4dbvnAxs2MiljTVJSWF94FW5Wy//rlsYt05heBwwn5QYGt3T+feZ\n",
       "3F/8LXbzgfluLLde52RgfU9iJOk14evkbu3CcKN/Qu4vdmsVfheREwOwoE03n9ROsiD/NnfPFX4X\n",
       "RVC//2A2bDkZmK2iXCIZQ0oKa5FKJ8Xf8bm/0Y0v/I4nlUzWzL+NcbB8cng+/xb7i7/N3Qstfpv7\n",
       "FzrhWIOTgVkN5buxxrXpxhb6xxSGxzYNjykMj2nRjQZeBBZSSBCF3+b+5u7FFv3F34G6haTS1CK3\n",
       "19SDk4FZn8olmjVYPkGs0TRudGG4VX+jG10Y16p/dIvxq+duMSkxvNjit9G/qKl/Rd3ipv7FA4xf\n",
       "3Ga+dt3S4ViicjIws8rkhLR6oWskjGL/qKb+4ryjBugvzjuq8DtqBeOa+5s7gJdYPkEMNPzSSvw2\n",
       "d+3GN3dLBjFtSYtx90awKP0tnAzMzAYtvyJ+JMuSQ7G/eVzzfO2Gi/0jmpYdsYL+RjfQfCNazDcC\n",
       "eHsEj6T9Gvq5c+RQFlpVkqYCp5N25MyI+EoVcZhZf8pPxC8hVVsZsFrZG5Q0AvgWMJX00rTDJW1b\n",
       "dhy9QtKUqmOoCx+LZXwslvGx6IzSkwGwK3B/RPw1IhYDPwEOrCCOXjGl6gBqZErVAdTIlKoDqJEp\n",
       "VQcwHFSRDCaR3njZ8GgeZ2ZmFakiGdSvxdrMrM+VfjeRpN2BaRExNQ+fBCwtNiJLcsIwMxuCnrm1\n",
       "VNJI0ucW30J65/2NwOERcXepgZiZ2ctKv7U0Il6S9Ang16RbS7/nRGBmVq1aPnRmZmblqqIBeUCS\n",
       "pkq6R9J9kj5bdTxlkrSJpGsk3Snpz5L+OY9fV9KVku6VdIWkCVXHWgZJIyTdKunSPNyvx2GCpPMl\n",
       "3S3pLkm79fGxOCn//7hD0rmSRvfLsZB0lqQ5ku4ojGu77/lY3ZfPp/uvaP21SgZ+II3FwLERsR2w\n",
       "O/DxvP8nAldGxFbA1Xm4HxwD3MWyO9D69Th8A7gsIrYFXg/cQx8eC0mbAUcDO0XE9qRq5sPon2Nx\n",
       "NuncWNRy3yW9FjiUdB6dCnxb0oDn+1olA/r8gbSImB0Rt+X+Z4G7Sc9gvBs4J892DnBQNRGWR9LG\n",
       "wDuAM4HG3RH9eBzWBvaOiLMgtblFxDP04bEgfVhoMTA234gylnQTSl8ci4i4lvTlvqJ2+34gcF5E\n",
       "LI6IvwL3k86vbdUtGfiBtCxfBe0I3ABMjIg5edIc0reDh7uvAyfAcu/J78fjsDnwN0lnS/qTpDMk\n",
       "jaMPj0VEPAV8FZhJSgJPR8SV9OGxKGi37xuRzp8NKzyX1i0ZuDUbkDQeuAA4JiIWFKdFavEf1sdJ\n",
       "0gHA3Ii4lWWlguX0w3HIRgI7Ad+OiJ1IX0VbrhqkX46FpC2ATwGbkU524yW9vzhPvxyLVgax7wMe\n",
       "l7olg8eATQrDm7B8dhv2JI0iJYIfRsTFefQcSRvk6RsCc6uKryRvAt4t6SHgPGBfST+k/44DpH//\n",
       "j0bETXn4fFJymN2Hx+KNwHUR8WREvARcCOxBfx6Lhnb/J5rPpRvncW3VLRncDGwpaTNJq5MaQC6p\n",
       "OKbSSBLwPeCuiDi9MOkS4IjcfwRwcfOyw0lEnBwRm0TE5qQGwt9ExAfos+MAqR0JeETSVnnUfsCd\n",
       "wKX02bEgNZzvLmlM/r+yH+kGg348Fg3t/k9cAhwmaXVJmwNbkh7wbS8iatUBbyc9oXw/cFLV8ZS8\n",
       "73uR6shvA27N3VRgXeAq4F7gCmBC1bGWeEz2AS7J/X15HIA3ADcBM0hXw2v38bH4DCkZ3kFqMB3V\n",
       "L8eCVEqeRfqk5yPAkQPtO3ByPo/eA7xtRev3Q2dmZla7aiIzM6uAk4GZmTkZmJmZk4GZmeFkYGZm\n",
       "OBmYmRlOBjZMSXo2/06WdHiH131y0/AfOrl+syo4Gdhw1XiAZnPgfSuzYH4j5kBOWm5DEXuuzPrN\n",
       "6sjJwIa7U4G980dyjpG0mqTTJN0oaYakDwNImiLpWkm/AP6cx10s6eb8oaGj87hTgTF5fT/M4xql\n",
       "EOV13yHpdknvLax7uqSf5w/U/KgRnKRT88daZkg6rdQjY1ZQ+jeQzUr2WeDTEfEugHzyfzoidpU0\n",
       "Gvi9pCvyvDsC20XEw3n4yIiYJ2kMcKOk8yPiREkfj4gdC9tolEL+nvTqiNcD6wM3SfpdnrYD6UMj\n",
       "jwN/kLQn6TUBB0XENjm2tbqw/2aD4pKBDXfNr8DeH/igpFuBP5Le7fKaPO3GQiIAOEbSbcD1pDdA\n",
       "brmCbe0FnBvJXOC3wC6kZHFjRMyK9P6X24DJwNPAQknfk3Qw8MKQ99JsFTkZWD/6RETsmLstIuKq\n",
       "PP65xgySpgBvAXaPiB1ILw1cYwXrDV6ZfBqlhhcL45YAoyJiCenrU+cDBwCXD2VnzDrBycCGuwXA\n",
       "moXhXwMfazQSS9pK0tgWy60FzIuIhZK2IX2TumFxm0bma4FDc7vE+sCbSa8NbvmBnvzFsgkR8Svg\n",
       "OFIVk1kl3GZgw1XjinwGsCRX95wN/AfpS1l/yu/EnwscnOcvvsL3cuCjku4ivVL9+sK0/wZul3RL\n",
       "pO8sBEBQ3edZAAAAVElEQVREXCRpj7zNAE6IiLmStuWVX5kKUpL6haQ1SAnj2I7sudkQ+BXWZmbm\n",
       "aiIzM3MyMDMznAzMzAwnAzMzw8nAzMxwMjAzM5wMzMwMJwMzMwP+B+Ha93Rhg8NBAAAAAElFTkSu\n",
       "QmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107d421d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(len(lr.cost_)), lr.cost_)\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Cost')\n",
    "plt.title('Stochastic Gradient Descent')\n",
    "#plt.text(40, 45, 'eta = 0.01')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
